<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
    <head>
        <title>RVM : Testing the RVM</title>
	    <link rel="stylesheet" href="styles/site.css" type="text/css" />
        <META http-equiv="Content-Type" content="text/html; charset=UTF-8">	    
    </head>

    <body>
	    <table class="pagecontent" border="0" cellpadding="0" cellspacing="0" width="100%" bgcolor="#ffffff">
		    <tr>
			    <td valign="top" class="pagebody">
				    <div class="pageheader">
					    <span class="pagetitle">
                            RVM : Testing the RVM
                                                    </span>
				    </div>
				    <div class="pagesubheading">
					    This page last changed on May 29, 2009 by <font color="#0050B2">pizlo</font>.
				    </div>

				    <p>Jikes RVM includes a testing framework for running functional and performance tests and it also includes a number of actual tests. See <a href="External Test Resources.html" title="External Test Resources">External Test Resources</a> for details or downloading prerequisites for the tests. The tests are executed using an Ant build file and produce results that conform to the definition below. The results are aggregated and processed to produce a high level report defining the status of Jikes RVM.</p>

<p>The testing framework was designed to support continuous and periodical execution of tests. A "<em>test-run</em>" occurs every time the testing framework is invoked. Every "<em>test-run</em>" will execute one or more "<em>test-configuration</em>"s. A "<em>test-configuration</em>" defines a particular build "<em>configuration</em>" (See <a href="Configuring the RVM.html" title="Configuring the RVM">Configuring the RVM</a> for details) combined with a set of parameters that are passed to the RVM during the execution of the tests. i.e. a particular "<em>test-configuration</em>" may pass parameters such as <tt>&#45;X:aos:enable_recompilation=false &#45;X:aos:initial_compiler=opt &#45;X:irc:O1</tt> to test the Level 1 Opt compiler optimizations.</p>

<p>Every "<em>test-configuration</em>" will execute one or more "<em>group</em>"s of tests. Every "<em>group</em>" is defined by a Ant build.xml file in a separate sub-directory of <tt>$RVM_ROOT/testing/tests</tt>. Each "<em>test</em>" has a number of input parameters such as the classname to execute, the parameters to pass to the RVM or to the program. The "<em>test</em>" records a number of values such as execution time, exit code, result, standard output etc. and may also record a number of statistics if it is a performance test.</p>

<p>The project includes several different types of &#95;test run_s and the description of each the test runs and their purpose is given in <a href="Test Run Descriptions.html" title="Test Run Descriptions">Test Run Descriptions</a>.</p>

<div class='panelMacro'><table class='noteMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="images/icons/emoticons/warning.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td><b>Note</b><br />
<p>The <a href="Using buildit.html" title="Using buildit">buildit</a> script provides a fast and easy way to build and the system.&nbsp; The script is simply a wrapper around the mechanisms described below.</p></td></tr></table></div>


<h2><a name="TestingtheRVM-AntProperties"></a>Ant Properties</h2>

<p>There is a number of ant properties that control the test process. Besides the properties that are already defined in <a href="Building the RVM.html" title="Building the RVM">Building the RVM</a> the following properties may also be specified.</p>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'> Property </th>
<th class='confluenceTh'> Description </th>
<th class='confluenceTh'> Default </th>
</tr>
<tr>
<td class='confluenceTd'> test-run.name </td>
<td class='confluenceTd'> The name of the <em>test-run</em>. The name should match one of the files located in the <a href="http://svn.sourceforge.net/viewvc/jikesrvm/rvmroot/trunk/build/test-runs/">build/test-runs/</a> directory minus the '.properties' extension. </td>
<td class='confluenceTd'> pre-commit </td>
</tr>
<tr>
<td class='confluenceTd'> results.dir </td>
<td class='confluenceTd'> The directory where Ant stores the results of the test run. </td>
<td class='confluenceTd'> <tt>${jikesrvm.dir}/results</tt> </td>
</tr>
<tr>
<td class='confluenceTd'> results.archive </td>
<td class='confluenceTd'> The directory where Ant gzips and archives a copy of test run results and reports. </td>
<td class='confluenceTd'> <tt>${results.dir}/archive</tt> </td>
</tr>
<tr>
<td class='confluenceTd'> send.reports </td>
<td class='confluenceTd'> Define this property to send reports via email. </td>
<td class='confluenceTd'> (Undefined) </td>
</tr>
<tr>
<td class='confluenceTd'> mail.from </td>
<td class='confluenceTd'> The from address used when emailing report. </td>
<td class='confluenceTd'> jikesrvm-core@lists.sourceforge.net </td>
</tr>
<tr>
<td class='confluenceTd'> mail.to </td>
<td class='confluenceTd'> The to address used when emailing report. </td>
<td class='confluenceTd'> jikesrvm-regression@lists.sourceforge.net </td>
</tr>
<tr>
<td class='confluenceTd'> mail.host </td>
<td class='confluenceTd'> The host to connect to when sending mail. </td>
<td class='confluenceTd'> localhost </td>
</tr>
<tr>
<td class='confluenceTd'> mail.port </td>
<td class='confluenceTd'> The port to connect to when sending mail. </td>
<td class='confluenceTd'> 25 </td>
</tr>
<tr>
<td class='confluenceTd'> &lt;configuration&gt;.built </td>
<td class='confluenceTd'> If set to true, the test process will skip the build step for specified configurations. For the test process to work the build must already be present. </td>
<td class='confluenceTd'> (Undefined) </td>
</tr>
<tr>
<td class='confluenceTd'> skip.build </td>
<td class='confluenceTd'> If defined the test process will skip the build step for all configurations and the javadoc generation step. For the test process to work the build must already be present. </td>
<td class='confluenceTd'> (Undefined) </td>
</tr>
<tr>
<td class='confluenceTd'> skip.javadoc </td>
<td class='confluenceTd'> If defined the test process will skip the javadoc generation step. </td>
<td class='confluenceTd'> (Undefined) </td>
</tr>
</tbody></table>

<h2><a name="TestingtheRVM-Definingatestrun"></a>Defining a test-run</h2>

<p>A <em>test-run</em> is defined by a number of properties located in a property file located in the <a href="http://svn.sourceforge.net/viewvc/jikesrvm/rvmroot/trunk/build/test-runs/">build/test-runs/</a> directory.</p>

<p>The property <em>test.configs</em> is a whitespace separated list of <em>test-configuration</em> "tags". Every tag uniquely identifies a particular <em>test-configuration</em>. Every <em>test-configuration</em> is defined by a number of properties in the property file that are prefixed with <em>test.config.&lt;tag&gt;.</em> and the following table defines the possible properties.</p>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'> Property </th>
<th class='confluenceTh'> Description </th>
<th class='confluenceTh'> Default </th>
</tr>
<tr>
<td class='confluenceTd'> tests </td>
<td class='confluenceTd'> The names of the test groups to execute. </td>
<td class='confluenceTd'> None </td>
</tr>
<tr>
<td class='confluenceTd'> name </td>
<td class='confluenceTd'> The unique identifier for <em>test-configuration</em>. </td>
<td class='confluenceTd'> "" </td>
</tr>
<tr>
<td class='confluenceTd'> configuration </td>
<td class='confluenceTd'> The name of the RVM build configuration to test. </td>
<td class='confluenceTd'> &lt;tag&gt; </td>
</tr>
<tr>
<td class='confluenceTd'> target </td>
<td class='confluenceTd'> The name of the RVM build target. This can be used to trigger compilation of a profiled image </td>
<td class='confluenceTd'> "main" </td>
</tr>
<tr>
<td class='confluenceTd'> mode </td>
<td class='confluenceTd'> The test mode. May modify the way test groups execute. See individual groups for details. </td>
<td class='confluenceTd'> "" </td>
</tr>
<tr>
<td class='confluenceTd'> extra.args </td>
<td class='confluenceTd'> Extra arguments that are passed to the RVM. </td>
<td class='confluenceTd'> "" </td>
</tr>
<tr>
<td class='confluenceTd'> extra.rvm.args </td>
<td class='confluenceTd'> Extra arguments that are passed to the RVM. These may be varied for different runs using the same image. </td>
<td class='confluenceTd'> "" </td>
</tr>
</tbody></table>
<div class='panelMacro'><table class='infoMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="images/icons/emoticons/information.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td><b>Note</b><br />
<p>The order of the test-configurations in <em>test.configs</em> is the order that the test-configurations are tested. The order of the groups in <em>test.config.&lt;tag&gt;.test</em> is the order that the tests are executed.</p></td></tr></table></div>
<p>The simplest <em>test-run</em> is defined in the following figure. It will use the build configuration "<em>prototype</em>" and execute tests in the "<em>basic</em>" group.</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>build/test-runs/simple.properties</b></div><div class="preformattedContent panelContent">
<pre>test.configs=prototype
test.config.prototype.tests=basic
</pre>
</div></div>
<p>The test process also expands properties in the property file so it is possible to define a set of tests once but use them in multiple test-configurations as occurs in the following figure. The groups basic, optests and dacapo are executed in both the prototype and prototype-opt test\configurations.</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedHeader panelHeader" style="border-bottom-width: 1px;"><b>build/test-runs/property-expansion.properties</b></div><div class="preformattedContent panelContent">
<pre>test.set=basic optests dacapo
test.configs=prototype prototype-opt
test.config.prototype.tests=${test.set}
test.config.prototype-opt.tests=${test.set}
</pre>
</div></div>

<h3><a name="TestingtheRVM-TestSpecificParameters"></a>Test Specific Parameters</h3>

<p>Each test can have additional parameters specified that will be used by the test infrastructure when starting the Jikes RVM instance to execute the test. These additional parameters are described in the following table.</p>
<table class='confluenceTable'><tbody>
<tr>
<th class='confluenceTh'> Parameter </th>
<th class='confluenceTh'> Description </th>
<th class='confluenceTh'> Default Property </th>
<th class='confluenceTh'> Default Value </th>
</tr>
<tr>
<td class='confluenceTd'> initial.heapsize </td>
<td class='confluenceTd'> The initial size of the heap. </td>
<td class='confluenceTd'> ${test.initial.heapsize} </td>
<td class='confluenceTd'> ${config.default-heapsize.initial} </td>
</tr>
<tr>
<td class='confluenceTd'> max.heapsize </td>
<td class='confluenceTd'> The initial size of the heap. </td>
<td class='confluenceTd'> ${test.max.heapsize} </td>
<td class='confluenceTd'> ${config.default-heapsize.maximum} </td>
</tr>
<tr>
<td class='confluenceTd'> max.opt.level </td>
<td class='confluenceTd'> The maximum optimization level for the tests or an empty string to use the Jikes RVM default. </td>
<td class='confluenceTd'> ${test.max.opt.level} </td>
<td class='confluenceTd'> "" </td>
</tr>
<tr>
<td class='confluenceTd'> processors </td>
<td class='confluenceTd'> The number of processors to use for garbage collection for the test or 'all' to use all available processors. </td>
<td class='confluenceTd'> ${test.processors} </td>
<td class='confluenceTd'> all </td>
</tr>
<tr>
<td class='confluenceTd'> time.limit </td>
<td class='confluenceTd'> The time limit for the test in seconds. After the time limit expires the Jikes RVM instance will be forcefully terminated. </td>
<td class='confluenceTd'> ${test.time.limit} </td>
<td class='confluenceTd'> 1000 </td>
</tr>
<tr>
<td class='confluenceTd'> class.path </td>
<td class='confluenceTd'> The class path for the test. </td>
<td class='confluenceTd'> ${test.class.path} </td>
<td class='confluenceTd'>&nbsp;</td>
</tr>
<tr>
<td class='confluenceTd'> extra.args </td>
<td class='confluenceTd'> Extra arguments that are passed to the RVM. </td>
<td class='confluenceTd'> ${test.rvm.extra.args} </td>
<td class='confluenceTd'> "" </td>
</tr>
<tr>
<td class='confluenceTd'> exclude </td>
<td class='confluenceTd'> If set to true, the test will be not be executed. </td>
<td class='confluenceTd'>&nbsp;</td>
<td class='confluenceTd'> "" </td>
</tr>
</tbody></table>
<p>To determine the value of a test specific parameters, the following mechanism is used;</p>
<ol>
	<li>Search for one of the the following ant properties, in order.
	<ol>
		<li>test.config.&lt;build-configuration&gt;.&lt;group&gt;.&lt;test&gt;.&lt;parameter&gt;</li>
		<li>test.config.&lt;build-configuration&gt;.&lt;group&gt;.&lt;parameter&gt;</li>
		<li>test.config.&lt;build-configuration&gt;.&lt;parameter&gt;</li>
		<li>test.config.&lt;build-configuration&gt;.&lt;group&gt;.&lt;test&gt;.&lt;parameter&gt;</li>
		<li>test.config.&lt;build-configuration&gt;.&lt;group&gt;.&lt;parameter&gt;</li>
	</ol>
	</li>
	<li>If none of the above properties are defined then use the parameter that was passed to the &lt;rvm&gt; macro in the ant build file.</li>
	<li>If no parameter was passed to the &lt;rvm&gt; macro then use the default value which is stored in the "Default Property" as specified in the above table. By default the value of the "Default Property" is specified as the "Default Value" in the above table, however a particular build file may specify a different "Default Value".</li>
</ol>


<h3><a name="TestingtheRVM-Excludingtests"></a>Excluding tests</h3>

<p>Sometimes it is desirable to exclude tests. The test exclusion may occur as the test is known to fail on a particular target platform, build configuration or maybe it just takes too long. To exclude a test, you must define the test specific parameter "exclude" to true either in .ant.properties or in the test-run properties file.</p>

<p>i.e. At the time of writing the Jikes RVM does not fully support volatile fields and as a result th test named "TestVolatile" in the "basic" group will always fail. Rather than being notified of this failure we can disable the test by adding a property such as "test.config.basic.TestVolatile.exclude=true" into test-run properties file.</p>

<h2><a name="TestingtheRVM-Executingatestrun"></a>Executing a test-run</h2>

<p>The tests are executed by the Ant driver script <em>test.xml</em>. The <em>test-run.name</em> property defines the particular test-run to execute and if not set defaults to "<em>sanity</em>". The command <tt>ant &#45;f test.xml &#45;Dtest-run.name=simple</tt> executes the test-run defined in <em>build/test-runs/simple.properties</em>. When this command completes you can point your browser at <tt>${results.dir}/tests/${test-run.name}/Report.html</tt> to get an overview on test run or at <tt>${results.dir}/tests/${test-run.name}/Report.xml</tt> for an xml document describing test results.</p>

				    
                    			    </td>
		    </tr>
	    </table>
	    <table border="0" cellpadding="0" cellspacing="0" width="100%">
			<tr>
				<td height="12" background="http://docs.codehaus.org/images/border/border_bottom.gif"><img src="images/border/spacer.gif" width="1" height="1" border="0"/></td>
			</tr>
		    <tr>
			    <td align="center"><font color="grey">Document generated by Confluence on Jun 10, 2009 09:12</font></td>
		    </tr>
	    </table>
    </body>
</html>